Virtual application execution system and method

ABSTRACT

A virtual execution environment for software applications and other code objects. In one implementation, the present invention allows for virtual execution of applications by transparently capturing the rendered output of natively executed applications, and delivering the output to other processes on the same platform or to remote devices in a platform agnostic manner.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 11/166,663 now U.S. Pat. No. 7,950,026 with anissue date of May 24, 2011, and entitled “Virtual Application ExecutionSystem and Method,” which claims priority from U.S. Provisional PatentApplication Ser. No. 60/583,182 filed Jun. 24, 2004, which isincorporated herein by reference for all purposes.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to computer software and, moreparticularly to methods, apparatuses and systems directed to virtualexecution environments for software applications.

BACKGROUND OF THE INVENTION

The emergence of distributed computing environments and associatedtechnologies, such as web services, wireless networks, and the likeposes a genuine challenge to the traditional deployment of softwareapplications. What is missing between the two poles of locally-deployed,compiled applications and distributed applications operating over anetwork has been a cohesive framework that allows developers to deploysoftware applications with the complexity and richness of compiledexecutables, while retaining the platform independence and thinness ofcontent and applications for web browsers or other thin hostingenvironments.

SUMMARY OF THE INVENTION

The present invention provides methods, apparatuses and systemsdirected, in one implementation, to a virtual execution environment forsoftware applications and other code objects or data blocks. In oneimplementation, the present invention allows for virtual execution ofapplications by transparently capturing the rendered output of nativelyexecuted applications, and delivering the output to other processes onthe same platform or to remote devices in a platform agnostic manner.The present invention allows for an unprecedented mixing of thinapplications in a distributed computing environment, such as apeer-to-peer system, with a rich API that makes no differentiationbetween the client and server nodes and has no dependency on theunderlying operating system. Implementations of the present inventionbridge the power of compiled applications with the thinness of webmedia. In some implementations, the present invention allows for theability to run content as meta-applications on devices without theruntime engine being present locally on the client device. The presentinvention, in one implementation, also features mechanisms thatfacilitate the dissemination, across one or more remote hosts, ofruntime engine code that incorporate the functionality described herein.The present invention can be applied in a variety of contexts includinginteractive systems-such as, computer gaming systems, three-dimensionalgraphical user interfaces, messaging systems, etc.—or passive• systems,such as multi-media presentations, animations, and the like. The presentinvention also has application to stand-alone computing devices, ornetworked computing systems. These and other objects of the presentinvention will become apparent from the drawings and followingdescription.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A, 1B, and 1C are a flow chart diagram setting forth a method,according to one implementation of the present invention, directed tothe hosting of an executable or native application within the context ofa browser or any other application that can render web media.

FIGS. 2A and 2B are a flow chart diagram showing a method, according toone implementation of the present invention, directed to the skinning ofbrowsers customized to one or more URLs.

FIG. 3 is a flow chart diagram showing the dissemination of a runtimestub, according to an implementation of the present invention.

FIG. 4 is a flow chart diagram showing the initiation of a shared,virtual hosting session between a runtime engine and a thin client.

FIG. 5 is a functional block diagram setting forth one possible systemarchitecture for a general purpose computer on which embodiments of thepresent invention maybe executed.

FIG. 6 is graphical user interface of a browser that includes thecaptured output of an application rendered therein.

DESCRIPTION OF PREFERRED EMBODIMENTS

As described herein, the virtual application execution system can beconfigured to accomplish a variety of functions and can operate as anindependent process or within the context of myriad applications, suchas games, browser clients, instant messaging clients, email clients, andthe like. In one implementation, the virtual application executionsystem is a software application executed on a general-purpose computer.The virtual application execution system may be implemented in a varietyof forms. For example, it may be distributed as a stand-aloneapplication, or as a plug-in to another application, such as a browser,email, or instant messaging client. In addition, the software may bedistributed as a download or on a computer readable medium, such as aCD, or DVD. The virtual application execution system functionalityaccording to the present invention can be incorporated into, and/oroperate with, a variety of computing platforms, such as game boxes,handheld devices, and the like.

In one implementation, the virtual application execution system is asoftware engine, embodied in a runtime library built using C/C++ andassembly language. Of course, other programming languages can be used.The virtual application execution engine can be distributed as aplug-in—such as a browser or IM client plug-in—, or as an independentexecutable. In one implementation, the virtual application executionengine comprises a virtual machine that interprets scripts, uses its ownvirtual desktop and file system (which can wrap around local native filestreams or raw socket connections) and includes a rendering engine thatperforms all rasterization (and in some embodiments, encryption andcompression). As discussed in more detail below, the virtual applicationexecution engine may also include other functionality, such as HTTP orweb server functionality, instant message (IM) functionality and thelike.

In one implementation, when content intended for the virtual applicationexecution engine is selected, such as when a user clicks on a link tothe content in a web page, the virtual application execution engine, ifnot already installed on the system, installs itself through a stubmechanism. The runtime stub manages all platform specific calls made bythe runtime engine. In one implementation, the remainder of the virtualapplication execution engine, above the stub layer, is self-containedand portable across a variety of platforms. In one implementation, theruntime stub is installed from a URL (in a web page, email, document, orinstant message) that references runtime content. The stub can bedistributed in a variety of forms, such as a web browser plug-in, anexecutable, media codec, QuickTime plug-in, Netscape plug-in, ActiveXcontrol, Shockwave Xtra, etc. In one implementation, the runtime stub isoperative to check the digital signatures for all remaining runtimelibraries and components that it loads.

As described herein, the virtual application execution engine allows forvirtual execution of applications by transparently launching andcapturing the rendered output of natively executed applications, anddelivering the output to other processes on the same platform or toremote devices in a platform agnostic manner (such as through an HTTPstream). For example, the virtual application execution engine can beconfigured to launch an executable within the context of an operatingsystem platform and render the output to a browser client executed onthe same platform. In one implementation, the output can be rendered toallow the user to interact with the virtually executed applicationwithin the context of the browser client as if it were a nativeembeddable object within a web page. In another implementation, thevirtual application execution engine can be configured to dynamicallyre-skin a browser client in response to a selected uniform resourcelocator (URL). In yet another embodiment, in one implementation, thevirtual application execution engine, when executed on a computingdevice, can be a node in a distributed computing environment, such as apeer-to-peer network, which can be used for data or file sharing, andvirtual hosting of applications for thin clients that do not have theruntime or other functionality to execute a desired application.

The virtual application execution engine, in one implementation, canoperate in two modes depending on the nature of the application to beexecuted. For example, the virtual application execution engine canoperate on an application package comprising, the binary executablecorresponding to an application, and a script (executed by the virtualmachine of the virtual application execution engine) defining theinstructions for launching the binary executable. In such animplementation, the virtual application execution engine silentlylaunches the binary executable as a native process, captures the outputand input channels to the binary executable, and renders and directs theoutput of the executable as desired. In a second mode, the applicationto be executed supports Microsoft's COM or other suitable interface. Inthis second mode, the virtual application execution engine launches theapplication as a COM process, captures the rendered output of the COMprocess, and directs• the rendered output as desired.

A. Virtual Execution of Software Applications or Executable Objects

As discussed herein, executables (*.exe) or other native applicationfiles and code objects can be launched into a process controlled by thevirtual application execution engine, and appear directly within awindow corresponding to the engine, or routed into another application,such as a frame within a web page. For example, implementations of thepresent invention can be configured to achieve the result illustrated inFIG. 6. That is, the hosting of an executable within the context of abrowser window. The virtual application execution engine, in oneimplementation, can capture a binary stream characterizing the displayoutput of an executable and re-route it. For example, the virtualapplication execution engine can unpack an application, and launch theexecutable silently, kidnapping its output for the above use using themethod shown in the code set forth below.

In implementations, where the virtual application execution engine isimplemented as a browser plug-in, native applications can be executedwithin the context of a browser. As FIG. 1 (comprising FIGS. 1A, 1B &1C) illustrates, local execution of a native application within thecontext of a browser can be triggered by a web page including standardobjects or embedded tags (102). In one implementation, the HTML tags setthe bounds of the executable to be embedded in the HTML page (104). AsFIG. 1 illustrates, the HTML parameters of the page may also specify thename of the executable to load and the URL of a binary manifest file(106). If the executable is stored locally on the computing system(108), the virtual application execution engine can authenticate theexecutable and libraries by comparing corresponding file signatures witha server-side registry of authorized applications (110). In oneimplementation, the virtual application execution engine authenticatesthe executable by matching one or more digital signatures in themanifest file with a trusted root authority (112).

If the executable is not stored locally, the virtual applicationexecution engine pulls, from the manifest file, the URLs for thepackaged files contain the executable and all dependent components(114), and downloads all necessary packed files in a background thread(116). As FIG. 1 illustrates, the virtual application execution enginethen authenticates the file signatures of the downloaded files using atrusted server or digital certificate(s) signed by a trusted rootauthority (118). If the files are not authenticated (120), the virtualapplication execution engine displays a warning dialog box, and canoptionally suppress the, launch of the application. Otherwise, thevirtual application execution engine places the files into a localsub-folder (122), and uses the manifest file or information in thepackage headers to build and populate sub-folders need by theexecutable, as well as modifying the system registry if necessary (124).

After authentication of the files, the virtual application executionengine launches the executable in a silent process within the context ofthe native operating system (126). In many operating systems (such asMicrosoft Windows®), a variety of objects are created to support thelaunched process, such as a parent or root window (identified by awindow handle), as well as window event loops or processes that scan foruser inputs (e.g., key strokes, mouse inputs, etc.) intended for theprocess. Together, the window and corresponding event loops are theinput/output interface to the launched process. In one implementation,the process is operative to render a display output as a binary datastream and provides the rendered output to the window created for it bythe operating system. As described herein, the virtual applicationexecution engine, in one implementation, intercepts the channel betweenthese interfaces and the underlying process to virtually execute thesubject executable within other contexts, such as within a browserclient. In one implementation, the virtual application execution enginelaunches the process off the edge of the display area with no visibletask bar, window or icon visible to the user. The virtual applicationexecution engine then scans the system windows of the native operatingsystem in a threaded loop as the executable code object is launched(128). During this loop, the virtual application execution engine testseach window on the operating system for an expected window name and aknown process identifier returned by the operating system when theprocess was launched (130). As FIG. 1 illustrates, if the virtualapplication execution engine finds a window that matches these twocriteria and is a root application window (132), it turns the windowinvisible to the user and stores the window handle corresponding to thewindow (134). Additional or alternative criteria that can be used toidentify the window corresponding to the silently launched processinclude thread handles, module names, class names, and the like. In oneimplementation, the virtual application execution engine changes atleast one window property (e.g., the procedure pointer in Microsoft OSs)created by the operating system when launching the process that allowsthe virtual application execution engine to capture the rendered outputof the process and divert it as desired, such as rendering it to anoff-screen buffer. For example, in Microsoft platforms, the virtualapplication execution engine changes the procedure pointer of thecaptured process from the window handle assigned by the operating systemto another procedure pointer created by the virtual applicationexecution engine to allow the virtual application execution engine tocontrol, render and process events for the captured application. As FIG.1 further illustrates, the virtual application execution engine, in oneimplementation, transforms the launched process' window into aborderless window and hooks into, or sub-classes, the window event loop(136) in order to intercept, as well as write, key strokes, mouse clicksand other inputs intended for the process. The source code set forthbelow illustrates how the virtual application execution engine launchesan executable and scans for a corresponding window created by theunderlying operating system.

EXE Virtual Hosting

//return window of a specific application process, Window containerclass with target process ID passed into this callback:

BOOL CALLBACK ScanProcessChildProc(HWND hWnd, LPARAM 1Param {  OSControl* OSW = (OSControl *)1Param;   DWORD dwProcessId;   if(!hWnd) return true;.   if (!::IsWindow(hWn)) return true;  GetWindowThreadProcessId(hWnd, &dwProcessId);   // first check theprocess matches   if (OSW->m_AppID != dwProcessId) return true;   //second, only get the root window   if (::GetParent(hWnd)) return true;  // third check the window title, ignoring if Name( ) length <=0 :   if(strlen(OSW->Name( ))>0)   {    char name[255];   ::GetWindowText(hWnd,name,255);    // if the name doesn't match, thenexit    if (!strcmp(OSW->Name( ),name)) return true;   }   // nativewindow handle stored back into the class if all tests.pass  OSW->os.window = hWnd;   return true; } // Create a kidnapped windowfrom a launched process created from the module at the specified pathEXEHost(const char *AppName,const char* CmdLine, const char* WndName =nil) ): CustomWindowClass( ) {   String WEXEAndCmdLine;   WEXEAndCmdLine= AppName;   // build path   WEXEAndCmdLine = “”;// optionallocalfolder;   WEXEAndCmdLine += AppName;   if (CmdLine != nil)  { WEXEAndCmdLine += “ ”;     WEXEAndCmdLine += CmdLine;   }   //launch process so it is hidden, and way off screen (since flickering mayoccur anyways, even if the hidden flag is true)   STARTUPINFO info; memset(&info,0,sizeof(info));  info.cb = sizeof(info); info.wShowWindow=SW_HIDE;  info.dwY = 9000;  info.dwX = 0;  info.dwFlags = STARTF_USESHOWWINDOW | STARTF_USEPOSITION;PROCESS_INFORMATION process; char buf[1024];  WEXEAndCmdLine.copy(buf,1024);   // wait in loop until the window forthe launched exe is found, then immediately hide its window and reattachit elsewhere including a web page frame  bool OK =CreateProcess(0,buf,0,0,false,DETACHED_PROCESS |INORMAL_PRIORITY_CLASS,0,0,&info,&process);   if (OK)   {   PostMessage(ParentWindow /*main window of the calling process*/,WM_NULL, 0, 0);    WaitForInputIdle( process.hProcess, INFINITE );   m_AppID = process.dwProcessId;    os.window = nil;    if (WndName)SetName(WndName);    else SetName(“”);    while (!os.window)    {   EnumWindows(ScanProcessChildProc, (LPARAM) this);    }   PostMessage(os.window, WM_NULL, 0, 0); } Synch( );   // window classfunction to set the window invisible, just in case the CreateProcessfunction failed to do so, as it sometimes does depending on theenvironment and type of application launched   SetVisible(false); }

As FIG. 1 shows, the virtual application execution engine processes themanifest file for scripts and code that may override the defaultbehavior of the executable parent window originally created by theoperating system and hidden by the virtual application execution engine(138). For example, the virtual application execution engine, consultinga script, can change the dimensions of the window created by the virtualapplication execution engine, background colors, scroll bars, etc. Inone implementation, the virtual application execution engine checks themanifest to determine if the executable can properly render to anoff-screen memory buffer, or if the executable was launched in a virtualoff-screen desktop or in the context of a hidden display device (140).If the executable can render to an off-screen buffer (142), the virtualapplication execution engine renders and manages the executable'soff-screen desktop shell or draws its context and rasterizes it into anoff-screen buffer (146). Otherwise, the virtual application executionengine removes the border of the root executable window of the capturedprocess and inserts it as a child window within another window managedby the virtual application execution engine (144). As FIG. 1 alsoillustrates, the virtual application execution engine uses thesub-classed event loop to propagate user interaction to the nativeprocess corresponding to the executable (148). That is, the virtualapplication execution engine receives user interaction (e.g., mouseclicks, key strokes, etc.) within the context of its window event loops,translates the event data as required (e.g., converting from 3D to 2Dspace, etc.), and writes it to the event loops of the process launchedfor the hosted executable. The virtual application execution engine thenrenders the embedded executable child window within a surface or spritelayer of the rendering context of the virtual application executionengine (150). For example, in one implementation, the resulting renderedoutput stored in the off-screen memory buffer, or the output renderedinto the child window, is placed inside the context of a browserapplication, or any other application that processed and launched theURL. The sub-classed (captured) process can then digest the userinteraction, the results of which the virtual application executionengine captures and renders within a destination display context such asa browser client window. Finally, when the plug-in is unloaded, thevirtual application execution engine terminates the hosted executableprocess before the plug-in window is destroyed.

As discussed above, other implementations are possible. For example, ifthe executable supports a COM or other suitable interface, the virtualapplication execution engine looks for a COM interface supported by theexecutable, and accesses the executable through its COM interface topotentially access additional information or other functionality notavailable through the normal event message pipeline. For example, thevirtual application execution engine can query the COM interface of aprocess to determine whether the COM interface can be used to render theoutput of the process to an off-screen buffer, get and receive eventmessages, and the like. The virtual application execution engine,depending on the properties of the COM process, can then capture itsoutput, in a manner similar to that discussed above, and direct theoutput as desired. Similar to the foregoing, the resulting outputrendered by the executable, in one implementation, is diverted into anoff-screen buffer and eventually blitted into the rendering context ofthe virtual application execution engine. Of course, other applicationinterfaces for external processes can be used, such as Java interfaces,.NET interfaces, Applescript interfaces, and the like.

B. Web Browser Skinning

In one implementation, the present invention facilitates the skinning ofbrowsers in a manner specifically configured or customized by a givenweb site. For example, Netscape used to give away themed browsers,having a customized skin, to promote a particular movie. The presentinvention takes this a step further, by automatically re-skinning thebrowser when a URL with HTML containing skin tags (or a URL matching apredetermined domain) is loaded (either through a link, or by typing theURL into the browser address bar). Further, when the URL is exited, thevirtual application execution engine restores the normal skin of thebrowser, and can maintain different skins for each browser instance. OnMicrosoft Windows® platforms, the virtual application execution enginecan use COM event sinks, or other external connections, in InternetExplorer® (IE) to capture the window handle of the browser, and pull theinner HTML control from the IE window, and place it in a new window withthe same dimensions as the normal browser using the URL's skinninginformation. The original browser window is hidden until the page isexited, at which point the inverse of the above is performed, with theHTML browser control being swapped back into the original frame. Thevirtual application execution engine can also leave the existing browserpage alone and achieve a cross fade or animated transition to the themedbrowser skin by taking a snapshot image of the browser window, launchinga completely new window with this image over the browser and then hidingthe browser window. Since the browser's display is being pulled from animage map maintained by the virtual application execution engine,already skinned and framed, the rendered HTML control can directly beblended into the image map, creating a seamless transition from thenormal as browser to the themed variation rendered by the virtualapplication execution engine. Furthermore, the virtual applicationexecution engine automatically propagates these enhancements (asplug-ins) within the client web browser, allowing the browser to 1)transparently host clipped executables (above) in a web page (as if theywere a native browser plug-in), and 2) re-skin the browser's borders,icons and buttons based on the currently viewed URL.

FIG. 2 illustrates a method directed to skinning a browser with customcontent as implemented by a browser plug-in that is triggered by anembedded tag or other object in an HTML page. As FIG. 2 illustrates, thebrowser itself can launch the plug-in virtual application executionengine upon detection of an embedded tag or component in an HTML page(203). Alternatively, the virtual application execution engine canalready be installed and running as a background process that connectsto system-wide browser COM or DOM (Document Object Model) events todetermine whether to launch the browser skinning functionality describedherein based on the URL currently loaded by any of the foregoing browserinstances (202). In this mode of implementation, the virtual applicationexecution engine, in one implementation, determines whether the URL,loaded by any current browser instance, matches an entry in a registry,which may be hosted by a remote server or stored locally (204). If amatch is found, the virtual application execution engine launches thefunctionality described herein.

In one implementation, the virtual application execution engine movesthe existing browser window in front of all other windows, if necessary,and captures its screen contents in an off-screen buffer (206). Thevirtual application execution engine then creates a new window (in oneimplementation, with no taskbar component), that matches the dimensionsand contents of the original browser window and displays it over thebounds of the original browser window (208). The virtual applicationexecution engine, in one implementation, hides the original browserwindow and taskbar icon as the new window is seamlessly swapped for itand becomes visible (210).

The virtual application execution engine, in one implementation, createsa child as window within the rendering context of the virtualapplication execution engine using standard system calls, or calls to anHTML rendering library that draws the target web page contents to anoff-screen buffer (212). As discussed above, the virtual applicationexecution engine may also use hidden display or virtual desktop graphicscontexts to achieve the effect of rendering the web page to anoff-screen buffer. Of course, other graphics contexts controlled orcontrollable by the virtual application execution engine can be used.The virtual application execution engine then subclasses and kidnaps theHTML-rendered child window and renders it within the newly createdwindow managed by the virtual application execution engine (214). Inanother implementation, subclassing of the event and message loops canbe replaced or enhanced, if the browser supports a COM or similarexternal application interface. The virtual application executionengine, if the target web page is rendered to an off-screen buffer, thenblits the rendered page from the off-screen buffer to the renderingcontext created by the virtual application execution engine (216). Ifthe rendered target page is placed in a virtual desktop graphicscontext, the virtual application execution engine can either blit thetarget page directly from the virtual graphics context, or copy thetarget web page into the off-screen buffer, and then blit the page tothe rendering context of the virtual application execution engine. Forexample, the virtual application execution engine may include a3-dimensional rendering engine. In such an embodiment, the rendered pagemay be converted to a composited texture map and rendered on the surfaceof a desired 3-dimensional object. In another implementation, the pagemay be simply blitted into a 2-dimensional area surrounded by theskinned browser components created by the virtual application executionengine. As FIG. 2B illustrates, the virtual application execution enginethen renders the browser buttons, address bar, and frame and compositesthem with the rendered HTML contents of the page (218). To properlyhandle events, the virtual application execution engine creates andhandles callbacks from the rendered elements of the page so thatcommands can be sent to the captured browser window or HTML renderingcontext, and performs browser navigation functions as a user clickswithin the window or other context presented by the virtual applicationexecution engine (220). As FIG. 2B shows, the virtual applicationexecution engine handles events such as mouse clicks and keyboardstrokes (222) in this manner until the user navigates to a URL that isoutside the context defined for the browser skin (224). After the usernavigates outside this context (which can be specified in a script) orthe URL domain used to load the skin, the virtual application executionengine resizes the original, hidden browser window to the bounds of theskinned browser window and makes the original browser window visible,swapping it with the skinned browser window (226).

C. Shared Windows and Remote Application Hosting

In one implementation, elements maintained and rendered by the virtualapplication execution engine on a local client can be transmitted andrendered remotely onto other clients. This functionality can be used toaccomplish a variety of objectives, such as remote application hostingand/or shared window sessions. Shared windows and remote applicationhosting can be accomplished in two different manners. For example, aninstant messaging, chat, or email message can be used to virallydisseminate the virtual application execution engine stub to a remoteclient. That is, as discussed below, in one implementation, the virtualapplication execution engine can install its runtime on the remote host,and use it to establish a window for the shared application on theremote host. In another implementation, each virtual applicationexecution engine runtime; when executed on a computing device, canbecome a node in a peer-to-peer network, which can be used for data orfile sharing, and virtual hosting of applications for thin clients thatdo not have the runtime or other functionality to execute a desiredapplication. Accordingly, through a standard TCP/IP or other networkprotocol connection, two or more nodes can be linked to a shared virtualwindow, with each node viewing and interacting through a local viewportto the shared window space or context.

FIG. 3 illustrates a process flow implemented on a remote client inresponse to a message that includes a link to content corresponding tothe virtual application execution engine. For example, using an IMgateway (such as a TOC session socket connection, SMS messaging system,or server IM proxy) built into the virtual application execution engine,a user may send a single URL link (or other reference trigger) to theremote client in a message. When the remote client receives the message(302), the user may activate the link, causing the remote client tolaunch an appropriate client application (304), such as a web browser,microbrowser, etc., which transmits a request to the node identified inthe link (306). The target node may be a server or node different/remotefrom the virtual application execution engine that originally sent thelink, or may be the same (originating) node that originally sent thelink. In either implementation, the sent link is interpreted by anautomated web server or other server node in the network, when theremote client clicks on it. The server node, which receives the request,generates an HTTP (or other device agnostic) response (308) and sends itto the remote client. In one implementation, the HTTP or other responsecomprises an HTML stream that includes a script with logic triggersexecuted by the remote client that conditionally launch various processflows based on the presence of a virtual application execution engine onthe remote client and/or the ability to install an engine stub on theremote client.

As FIG. 3 illustrates, when the remote client receives and processes theHTTP response, the script (or other code object) in the response firstchecks for the presence of a virtual application execution engineinstalled on the remote client (310). If the virtual applicationexecution engine is present, the script activates the virtualapplication execution engine on the remote client (312), which thenoperates in a similar manner as that discussed above. For example, thevirtual application execution engine may launch as a browser plug-in onthe remote client, consult a .script and install an executable codeobject embedded in the HTTP response (see above). In addition, a sharedwindow or session may be established, as described in more detail below,where the virtual application execution engine on the remote client andthe originating host synchronize the events at each node in the sharedsession. If the virtual application execution engine is not installed onthe remote client, however, the script in the HTML received by theremote client first attempts to download and install an engine whichtransparently installs the virtual application execution engine runtime(316) and uses the generated HTML parameters passed to the browserplug-in (or other interface to the engine on the client) to host anexecutable code object and/or establish a shared window between eachclient. In one implementation, the runtime install can occur with just afew clicks from the remote user, even if it does not yet have thevirtual application execution engine installed before a first connectionis established. This allows the viral dissemination of both the virtualapplication execution engine runtime code and its managed content withalmost no installation or action required on the part of the remote user(e.g., one to click on the link sent in an IM, the other to accept theinstallation of the apparatus' browser plug-in, which can take the formof a stub ActiveX control on ape, or a QuickTime component or ShockwaveXtra on the Mac).

Assuming that the originating node and the remote client both haveruntime engines installed, the shared window session processes, such asevent and rendering synchronization, is greatly facilitated and can beimplemented using standard technologies, such as technologies in thenetwork gaming environment. For example, the remote client andoriginating host can send smaller representations of complex structures,such as the drawing context or transforms of objects or state renderingchanges. For example, one node may simply send position or vector datafor a given 3-dimensional object as opposed to having to sendinformation encapsulating the entire rasterization of the object on thelocal client.

As FIG. 3 illustrates, if the remote client does not support the enginestub (314), the script or other parameters in the HTTP response attemptsto find a node in the network environment, such as the originating node,that can virtually host the application and/or shared window session(318). In another implementation, however, the remote client may rely onfunctionality native to the original server's network environmentitself, such as a peer-to-peer network to identify a node that can hostthe application or shared window session. In one implementation, thisfunctionality could be triggered by a timeout at the remote client,after the application failed to execute within a threshold period oftime.

As discussed above, the virtual application execution engine can alsooperate to virtually host an application and/or establish a sharedwindow (in one implementation, rendered as an HTML page, including imagemaps, Javascript events and triggers, etc.) with thin clients thatcannot support the engine runtime stub. To support shared windowsessions, the virtual application execution engine can translateapplication events and logic, local to the hosting node, into an HTTPstream destined for the remote client. User interaction (such asJavascript events, image map events, and changes to text fields) at theremote client are transmitted back to the virtual application executionengine on the hosting node (in one implementation, through HTTPstreams), and translated into local events and written to the eventloop(s) of the shared process(es). In addition, some events on theremote client can be generated by the Javascript or other code in theHTTP stream, such as automatic refreshing of pages, timeouts, mouseevents, keyboard events, etc. Translation of the local events into HTTPstreams, for example, essentially results in abstraction of these eventsthat can be configured to achieve a variety of features and objectives.For example, an input device (e.g., a joystick, keyboard or other I/Odevice) expected by the process (executed on the hosting node) that doesnot physically exist on the remote client can be nevertheless simulated.For example, the virtual application execution engine can convert imagemap inputs from the remote client in a response HTTPstream to equivalentjoystick or keyboard input and write or transmit the converted inputs tothe process executed on the hosting node. The virtual applicationexecution engine can also convert any static or editable content in thehosted process (e.g., text, fields, sprites, arrays, labels, buttons,etc.) into equivalent HTML structures (e.g., tables, HTML layers, taggedobjects, canvases, scripts, and Java applets) for transmission to, aswell as consumption and synchronization by, the remote client. In oneimplementation, however, the virtual application execution engine cansimply transmit non-interactive images (such as JPEGs and GIFs) of therasterized application context on the host.

For example, a cell phone running a microbrowser client can connect toanother node that includes a virtual application execution engineruntime and a word processing or spreadsheet application. The virtualapplication execution engine runtime can create an interface to adocument with full event synchronization that works with themicrobrowser on the cell phone and transmit it to the cell phone. Forexample, the virtual application execution engine can be used to host aspreadsheet program (with support for COM or other external applicationinterfaces), as described above, capture the rendered output of thespreadsheet program, and create a page including the rendered output asan HTML table that has been translated from accessing the column and rowproperties of the spreadsheet through its COM or other externalinterface. A link to the page can then be transmitted to the cell phonein an instant message. The virtual application execution engine includesHTTP hosting and/or other server functionality thattranslates/transcodes and serves the page to the cell phone when thelink is activated. User interaction on the cell phone is transmitted tothe virtual application execution engine server in a manner similar tonormal HTTP sessions. The virtual application execution engine receivesthe input and translates it to a standard windows event and writes it tothe event loop of the local process corresponding to the spreadsheet.The resulting output can be rendered on the local node, as well astransmitted to the cell phone's microbrowser. Accordingly, the virtualapplication execution engine, in effect, translates the output of alocally executed process, such as a graphics context, events, etc. intoan HTML over HTTP stream. In addition, the remote user's input istransmitted back to the virtual application execution engine as HTTPmessages. For example, in one implementation, the resulting renderedoutput stored in the off-screen memory buffer may be converted into aJPEG, GIF or other suitable image file and displayed within the contextof a remote client. In another implementation, the rendered output canbe converted into an image map (or editable HTML field or table) anddisplayed within the context of a remote browser: The virtualapplication execution engine can also convert the mouse clicks on theimage map (or edits to input, fields or table objects) to correspondingevents and write them to the event loop corresponding to the hostprocess implementing the executable. The host process can then digestthe user interaction, the results of which the virtual applicationexecution engine captures, renders and transmits to the remote client.

FIG. 4 illustrates a process, according to one implementation of thepresent invention, executed by a hosting node in the situation where aremote client does not have an installed virtual application executionengine, and a shared window session is desired. In one implementation,the hosting node receives a host request (352), and decides whether toaccept the request (354). In one implementation, the hosting requestidentifies the application or code object to be executed and the remoteclient(s). The remote client may be identified based on a networkaddress, a hostname or any other suitable identifier. In addition, avariety of factors can be used in this determination, such as whetherthe instant host initiated the exchange with the remote client, the loadon the instant host, etc. If the host accepts the request, the virtualapplication execution engine on the host launches the identifiedapplication as a local process within the context of its operatingsystem, and captures the output of process, as discussed above, to anoff-screen buffer or in a kidnapped child window (whether visible orinvisible) (356). As FIG. 4 shows, the virtual application executionengine on the hosting node also transmits the resulting output to theremote client identified in the hosting request.

As FIG. 4 illustrates, the virtual application execution engine on thehosting node checks for local events (such as mouse clicks, keyboardstrokes, joystick movements, etc.) (358), converts the local events (asdiscussed above) and writes the events to the captured process (360). Inaddition, the virtual application execution engine also scans for remoteevent messages transmitted by the remote client(s) (362). As discussedin more detail below, the user on the remote client may interact withthe output displayed at the remote client. These interactions—which alsocan be mouse clicks, keyboard strokes, and the like—are transmitted inmessages, such as HTTP streams. The virtual application execution engineon the hosting node receives the event message on a socket or otherconnection mechanism, converts the remote events as appropriate for thenative, captured process, and writes them to the captured process (364).After the virtual application execution engine receives the resultingoutput of the process (366), it determines whether any state changesoccurred (368). If so, the virtual application execution engine on thehosting node translates the state changes and events to a formatappropriate to the remote client (370), and transmits a message to theremote client (372). The message contents can vary significantlydepending on the changes, the attributes of the hosted application,and/or the attributes of the remote client or client application. Forexample, the message contents may be include instructions describing thechanges themselves, or may be an entirely re-rendered image file or map.Given the foregoing, if the hosting node is selected solely to host theapplication for the remote client (and no shared window session isdesired), the virtual application execution engine omits the stepsdirected to checking for local events.

To facilitate initiation of a shared window session, in oneimplementation, the virtual application execution engine allows users tosign into their IM account (using public IM protocols like TOC, or aserver based IM gateway for protected protocols), subsequently creatingan object on the screen for each contact on the users' buddy lists, orgenerate a unique ID for the user (for use in a peer-to-peer or othernetwork structure) from one or more hardware identifiers on the localsystem, such as hard disk drive or CPU serial numbers. A given buddyobject represents a remote client, and when connected to a virtualapplication execution engine process or window by a user (such asdragging a buddy icon into a window or surface), makes the object'sprocess, logic and/or window shared between the local virtualapplication execution engine and the remote client (spawning a newwindow on the remote client if the shared content or process is notalready attached to a window or object on the remote client's machine).

D. Exemplary Computing Platform

Still further, in the embodiments described above, the present inventionis executed within the context of a software application installed on ageneral purpose computer. FIG. 5 illustrates, for didactic purposes, thesystem architecture of a general purpose computer according to oneimplementation of the present invention. The present invention, however;can be implemented on a wide variety of computer system architectures.In one embodiment, the above-described system architecture operates inconnection with computer hardware system 800 of FIG. 5. An operatingsystem manages and controls the operation of system 800, including theinput and output of data to and from the location diagram editingapplication, as well as other software applications (not shown). Theoperating system provides an interface, such as a graphical userinterface (GUI), between the user and the software applications beingexecuted on the system. According to one embodiment of the presentinvention, the operating system is the Windows® 95/98/NT/XP operatingsystem, available from Microsoft Corporation of Redmond, Wash. However,the present invention may be used with other conventional operatingsystems, such as the Apple Macintosh Operating System, available fromApple Computer Inc. of Cupertino, Calif., UNIX operating systems, LINUXoperating systems, and the like.

FIG. 5 illustrates one embodiment of a computer hardware system suitablefor use with the present invention. In the illustrated embodiment,hardware system 800 includes processor 802 and cache memory 804 coupledto each other as shown. Additionally, hardware system 800 includes highperformance input/output (I/O) bus 806 and standard I/O bus 808. Hostbridge 810 couples processor 802 to high performance I/O bus 806,whereas I/O bus bridge 812 couples the two buses 806 and 808 to eachother. Coupled to bus 806 are network/communication interface 824,system memory 814, and video memory 816. In turn, display device 818 iscoupled to video memory 816. Coupled to bus 808 are mass storage 820,keyboard and pointing device 822, and I/O ports 826. Collectively, theseelements are intended to represent a broad category of computer hardwaresystems, including but not limited to general purpose computer systemsbased on the Pentium® processor manufactured by Intel Corporation ofSanta Clara, Calif., as well as any other suitable processor.

The elements of computer hardware system 800 perform their conventionalfunctions known in the art. In particular, network/communicationinterface 824 is used to provide communication between system 800 andany of a wide range of conventional networks, such as an Ethernetnetwork, a wireless (e.g., IEEE 802.11) network, etc. Mass storage 820is used to provide permanent storage for the data and programminginstructions to perform the above described functions (implemented inthe system controller, whereas system memory 814 (e.g., DRAM) is used toprovide temporary storage for the data and programming instructions whenexecuted by processor 802. I/O ports 826 are one or more serial and/orparallel communication ports used to provide communication betweenadditional peripheral devices which may be coupled to hardware system800.

Hardware system 800 may include a variety of system architectures andvarious components of hardware system 800 may be rearranged. Forexample, cache 804 may be on-chip with processor 802. Alternatively,cache 804 and processor 802 may be packed together as a “processormodule”, with processor 802 being referred to as the “processor core”.Furthermore, certain implementations of the present invention may notrequire nor Include all of the above components. For example, theperipheral devices shown coupled to standard I/O bus 808 may be coupledto high performance I/O bus 806; in addition, in some implementationsonly a single bus may exist with the components of hardware system 800being coupled to the single bus. Furthermore, additional components maybe included in system 800, such as additional processors, storagedevices, or memories.

In one embodiment, the operations of the location diagram editingapplication are implemented as a series of software routines run byhardware system 800 of FIG. 5. These software routines comprise aplurality or series of instructions to be executed by a processor in ahardware system, such as processor 802. Initially, the series ofinstructions are stored on a storage device, such as mass storage 820.However, the series of instructions can be stored on any conventionalstorage medium, such as a diskette, CD-ROM, ROM, etc. Furthermore, theseries of instructions need not be stored locally, and could be receivedfrom a remote storage device, such as a server on a network, vianetwork/communication interface 824. The instructions are copied fromthe storage device, such as mass storage 820, into memory 814 and thenaccessed and executed by processor 802. In one implementation, thesesoftware routines are written in the C++ programming language and storedin compiled form on mass storage device 820. However, these routines maybe implemented in any of a wide variety of programming languages,including Visual Basic, Java, etc. In alternate embodiments, the presentinvention is implemented in discrete hardware or firmware. For example,an application specific integrated circuit (ASIC) could be programmedwith the above described functions of the present invention.

The invention has been explained with reference to specific embodiments.For example, the rendering engine and associated functionality can beimplemented on a variety of computing platforms, such as dedicatedgaming devices or boxes, handheld devices and the like. Otherembodiments will be evident to those of ordinary skill in the art. It istherefore not intended that the invention be limited

1. In a virtual application execution engine executing within acomputing environment comprising a processor, a memory connected to theprocessor and an operating system stored in the memory, a method forvirtually executing an executable code object, comprising receiving froma remote node, by the virtual application execution engine associatedwith a host node, a trigger to launch an executable code object locallywithin the context of the operating system; launching, by the virtualapplication execution engine, the executable code object locally as aprocess within the context of the operating system, wherein the processis operative to cause a root window to be created locally within thecontext of the operating system and render an output to the root window;intercepting, by the virtual application execution engine, the outputrendered by the process, wherein intercepting the output rendered by theprocess comprises iteratively scanning, in response to launching theexecutable code object and as the executable code object launches,system windows of the operating system against a set of expectedattributes to initially detect the root window, wherein the set ofexpected attributes comprises one or more of a process identifier, awindow name or an indication of a root window; and converting theintercepted output of the process to a first network protocol stream fordisplay on the remote node, wherein converting the intercepted outputcomprises overriding a default behavior of the root window of the hostnode by changing a property of the intercepted output for display on theremote node; and transmitting the first network protocol streamincluding the change in the property of the intercepted output to theremote node.
 2. The method of claim 1 wherein the first network protocolstream is an HTML over HTTP stream.
 3. The method of claim 1 furthercomprising receiving a second network protocol stream from the remotenode, wherein the second network protocol stream comprises an input forthe process; translating the input for the process from the secondnetwork protocol stream to a native input; and writing the native inputto the process.
 4. The method of claim 2 further comprising receiving anHTTP response to the HTML over HTTP data stream from the remote node;and converting the response to at least one event; and writing the atleast one event to the process.
 5. The method of claim 1 wherein theexecutable code object supports an external application interface, andwherein the method further comprises accessing additional functionalityassociated with the executable code object through the externalapplication interface.
 6. The method of claim 1 wherein the interceptingstep further comprises changing at least one attribute of the rootwindow to intercept the rendered output of the process.
 7. The method ofclaim 6 wherein the changing step comprises changing a first procedurepointer assigned to the root window by the operating system to a secondprocedure pointer.
 8. The method of claim 1, wherein the root windowfurther comprises an event loop, and wherein the method furthercomprises sub-classing the event loop to intercept input eventscorresponding to the process.
 9. An apparatus for virtually executing anapplication, comprising: a processor; a memory; an operating systemstored in the memory; and a virtual application execution engine,physically stored in the memory, comprising instructions operable tocause the processor and the apparatus to: receive from a remote node, bythe virtual application hosting execution engine associated with a hostnode, a trigger to launch an executable code object locally within thecontext of the operating system; launch, by the virtual applicationexecution engine, the executable code object locally as a silent processwithin the context of the operating system, wherein the silent processis operative to cause a root window to be created locally within thecontext of the operating system and render an output to the root window;intercept, by the application execution engine, the output rendered bythe silent process, wherein intercepting the output rendered by thesilent process comprises iteratively scanning, in response to launchingthe executable code object and as the executable code object launches,system windows of the operating system against a set of expectedattributes to initially detect the root window, wherein the set ofexpected attributes comprises one or more of a process identifier, awindow name or an indication of a root window; and convert theintercepted output of the silent process to a first network protocolstream for display on the remote node, wherein converting theintercepted output comprises overriding a default behavior of the rootwindow of the host node by changing a property of the intercepted outputfor display on the remote node; and transmit the first network protocolstream including the change in the property of the intercepted output tothe remote node.
 10. The apparatus of claim 9 further comprising anetwork interface, and wherein the virtual application execution enginecomprises a server application comprising instructions operable to causethe processor, the network interface, and the apparatus to communicatewith a remote host; and wherein the virtual application execution enginefurther comprises instructions operable to cause the processor and theapparatus to convert the rendered output to an HTML data stream over anHTTP session; and transmit the HTML data stream to the remote host. 11.The apparatus of claim 10 wherein the virtual application executionengine further comprises instructions operable to cause the processorand the apparatus to receive an HTTP response to the HTML data streamfrom the remote host; convert the response to at least one event; andwrite the at least one event to the process.
 12. The apparatus of claim9 wherein the virtual application execution engine further comprisesinstructions operable to cause the processor and the apparatus to changeat least one attribute of the window to intercept the rendered output ofthe silent process.
 13. The apparatus of claim 12 wherein the changingstep comprises changing a first procedure pointer assigned to the rootwindow by the operating system to a second procedure pointer.
 14. Theapparatus of claim 9, wherein the root window further comprises an eventloop, and wherein the instructions are further operable to cause theprocessor and the apparatus to sub-class the event loop to interceptinput events corresponding to the process.
 15. A computer programproduct, physically stored on a non-transitory machine-readable medium,comprising a virtual application execution engine for virtuallyexecuting a software application within the context of an operatingsystem, the virtual application execution engine comprising instructionsoperable to cause a programmable processor to: receive from a remotenode, by the virtual application execution engine associated with a hostnode, a trigger to launch an executable code object locally within thecontext of the operating system; launch, by the virtual applicationexecution engine, the executable code object locally as a silent processwithin the context of the operating system, wherein the silent processis operative to cause a root window to be created locally within thecontext of the operating system and render an output to the root window;intercept, by the virtual application execution engine, the outputrendered by the process, wherein intercepting the output rendered by theprocess comprises iteratively scanning, in response to launching theexecutable code object and as the executable code object launches,system windows of the operating system against a set of expectedattributes to initially detect the root window, wherein the set ofexpected attributes comprises one or more of a process identifier, awindow name or an indication of a root window; and convert theintercepted output of the process to a first network protocol stream fordisplay on the remote node, wherein converting the intercepted outputcomprises overriding a default behavior of the root window of the hostnode by changing a property of the intercepted output for display on theremote node; and transmit the first network protocol stream includingthe change in the property of the intercepted output to the remote node.16. The computer program product of claim 15 wherein the instructionsare further operable to cause the programmable processor to: communicatewith a remote host; and wherein the virtual application execution enginefurther comprises instructions operable to cause the processor and theapparatus to convert the rendered output to an HTML data stream over anHTTP session; and transmitting transmit the HTML data stream to theremote host.
 17. The computer program product of claim 16 wherein theinstructions are further operable to cause the programmable processor toreceive an HTTP response to the HTML data stream from the remote host;convert the response to at least one event; and write the at least oneevent to the process.
 18. The computer program product of claim 15wherein the instructions are further operable to cause the programmableprocessor and the apparatus to change at least one attribute of thewindow to intercept the rendered output of the silent process.
 19. Thecomputer program product of claim 18 wherein the changing step compriseschanging a first procedure pointer assigned to the root window by theoperating system to a second procedure pointer.
 20. The computer programproduct of claim 18, wherein the root window further comprises an eventloop, and wherein the instructions are further operable to cause theprocessor and the apparatus to sub-class the event loop to interceptinput events corresponding to the process.